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DETAILED ACTION 

1 . This action is responsive to communications: amendment filed 3/10/2005. 

2. Claims 1-5, 7-17, 19-29, 31-41, and 43-52 are pending in the case. Claims 1, 13, 25, and 
37 are independent claims. 

Claim Rejections - 35 USC §102 

3. The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the 
basis for the rejections under this section made in this Office action: 

A person shall be entitled to a patent unless - 

(e) the invention was described in (1) an application for patent, published under section 122(b), by another filed 
in the United States before the invention by the applicant for patent or (2) a patent granted on an application for 
patent by another filed in the United States before the invention by the applicant for patent, except that an 
international application filed under the treaty defined in section 351(a) shall have the effects for purposes of this 
subsection of an application filed in the United States only if the international application designated the United 
States and was published under Article 21(2) of such treaty in the English language. 

4. Claims 1-4, 7, 13-16, 19, 25-28, 31, and 44 are rejected under 35 U.S.C, 102(e) as 
being anticipated by Schena et al. (hereinafter "Schena"), US 6,448,979 Bl filed 1/25/1999. 

Regarding independent claims 1,13, and 25, Schena discloses creating a multimedia 
annotation for a paper document, the multimedia annotation representing at least one of an audio 
sound and a video clip in col. 2 lines 6-8. Schena discloses creating a first multimedia document 
by combining the paper document and the multimedia annotation such that the multimedia 
annotation can be extracted and decoded subsequently from the first multimedia document and 
played via a multimedia player, wherein the first multimedia document is generated as a part of 
reproducing the paper document via a document reproduction system in col. 1 line 61 - col. 2 
line 25. Schena reproduces a multimedia document by adding computer readable codes, such as 
bar codes, into the paper document. 
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Regarding dependent claims 2, 14, and 26, Schena discloses wherein the multimedia 
annotation is represented as a first bar code printed on the multimedia document in col. 3 lines 
58-63. 

Regarding dependent claims 3, 15, and 27, Schena discloses wherein the first bar code 
encodes an audio sound in col. 1 line 61 - col. 2 line 25. 

Regarding dependent claims 4, 16, and 28, Schena discloses wherein a location 
indicator associated with the multimedia annotation is placed on the first multimedia document, 
wherein the location indicator indicates where the multimedia annotation can be retrieved and 
played in col. 1 line 61 - col. 2 line 25. 

Regarding dependent claims 7, 19, and 31, Schena discloses wherein the first 
multimedia document is capable of being extracted by scanning and decoding the printed 
annotation of the paper document, and wherein the extracted multimedia annotation can be 
played via a multimedia player. 

Regarding dependent claim 44, Schena discloses wherein the multimedia document is a 
physical document having the first bar code printed thereon, wherein the audio sound can be 
extracted from the multimedia document by scanning and decoding the first bar code, and 
wherein the extracted audio sound is capable of being played via an audio player in col. 1 line 61 
- col. 2 line 25 and col. 3 line 64 - col. 4 line 6. 

Claim Rejections - 35 USC §103 
The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 
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(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in 
section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are 
such that the subject matter as a whole would have been obvious at the time the invention was made to a person 
having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the 
manner in which the invention was made. 

5. Claims 5, 17, and 29 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Schena et al. (hereinafter "Schena"), US 6,448,979 Bl filed 1/25/1999 in view of Gormish et 
al. (hereinafter "Gormish"), US 5,337,362 patented 8/9/1994. 

Regarding dependent claims 5, 17, and 29, Schena teaches wherein the location 
indicator comprises a first Uniform Resource Locator (URL) in col. 3 line 64 - col. 4 line 6. 
Schena does not teach representing a URL in an encrypted form. Gormish does teach 
representing a URL in an encrypted form in the abstract, fig. 8-9, and col. 2 lines 17-40. It 
would have been obvious to one of ordinary skill in the art at the time the invention was made to 
have combined Gormish into Schena to have created the claimed invention. It would have been 
obvious and desirable to have used the URL encryption teaching of Gormish to have enhanced 
the URL barcode of Schena so that access to the multimedia annotation would have been secure 
in the event the multimedia document is acquired by an unauthorized user. 

6. Claims 8-10, 20-22, 32-34, 37-41, and 45-52 are rejected under 35 U.S.C. 103(a) as 
being unpatentable over Schena et al. (hereinafter "Schena"), US 6,448,979 Bl filed 
1/25/1999 in view of Robinson et al. (hereinafter "Robinson"), "The Origami Project: 
Paper Interfaces to the World-Wide Web", 

[http://www.cLcam.ac.uk/Research/Origami/Origamil997f/index.html], submitted to 
Webnet 97 in November 1997, pages 1-6. 
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Regarding dependent claims 8, 20, and 32, Schena teaches creating a multimedia 
document composed of paper document and one or more multimedia annotations in col. 1 line 61 
- col. 2 line 25 and col. 3 line 64 - col. 4 line 6. Schena does not teach generating an image of 
the paper document, the image of the paper document being unconsciously captured during the 
reproduction of the paper document without user intervention. Schena does not teach creating a 
second multimedia document by combining the image of the paper document and the multimedia 
annotation, wherein the multimedia annotation is captured via an input device of the document 
reproduction system while the paper document is being reproduced via the document 
reproduction system. Schena does not teach storing the image of the paper document and the 
multimedia annotation in a storage. 

Robinson does teach generating an image of the paper document, the image of the paper 
document being unconsciously captured during the reproduction of the paper document without 
user intervention and creating a second multimedia document by combining the image of the 
paper document and the multimedia annotation, wherein the multimedia annotation is captured 
via an input device of the document reproduction system while the paper document is being 
reproduced via the document reproduction system in fig. 1, page 4 paragraphs 1-4. The 
document image acquired by the camera system may be combined with one or more other 
documents or multimedia annotations. The newly created document is the second multimedia 
document. Robinson does teach storing the image of the paper document and the multimedia 
annotation in a storage in fig. 1 and the "Registry" section in pages 2-3. Robinson's storage is 
called the registry, which stores the second multimedia document. 
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Regarding dependent claims 9, 21, and 33, Schena teaches wherein a multimedia 
document is represented as a second Uniform Resource Locator (URL) printed on the first 
multimedia document, and wherein the image of the paper document and the multimedia 
annotation is accessed with the second URL in col. 1 line 61 - col. 2 line 25 and col. 3 line 64 - 
col. 4 line 6. 

Regarding dependent claims 10, 22, and 34, Schena teaches wherein a bar code is used 
to represent a second URL in col 3 lines 58-63. 

Regarding independent claim 37, Schena teaches creating a paper document to be used 
with a multimedia annotation in col. 1 line 61 - col. 2 line 25. Schena teaches creating a 
multimedia annotation, the multimedia annotation representing at least one of an audio sound 
and a video clip in col. 2 lines 6-8. Schena teaches creating a multimedia document by 
combining the paper document and the multimedia annotation such that the multimedia 
annotation can be extracted and decoded subsequently from the first multimedia document and 
played via a multimedia player, wherein the storing and the combining are performed as a part of 
reproducing the paper document via a document reproduction system without user intervention 
in col. 1 line 61 - col. 2 line 25 and col. 3 line 64 - col. 4 line 6. Schena does not teach storing 
an image of the paper document. 

Robinson does teach storing and image of the paper document and the multimedia 
annotation in fig. 1, page 4 paragraphs 1-4. The document image acquired by the camera system 
may be combined with one or more other documents or multimedia annotations. Robinson 
stores the image in the registry. It would have been obvious to one of ordinary skill in the art at 
the time the invention was made to have combined Robinson into Schena to have created the 
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claimed invention. It would have been obvious and desirable to have used the paper document 
image acquisition teaching of Robinson to have acquired an image of the paper document so that 
the new multimedia document could have been merged and produced digitally as is taught by 
Robinson. 

Regarding dependent claim 38, Schena teaches creating a paper multimedia document 
by associating the multimedia annotation with the paper document in col. 1 line 61 - col. 2 line 
25 and col. 3 line 64 - col. 4 line 6. Schena does not teach creating an electronic multimedia 
document by associating the multimedia annotation with the image of the paper document, 
wherein the image of the paper document is captured while the paper document is being 
reproduced using the document reproduction system. Robinson does teach creating an electronic 
multimedia document by associating the multimedia annotation with the image of the paper 
document, wherein the image of the paper document is captured while the paper document is 
being reproduced using the document reproduction system in fig. 1, page 4 paragraphs 1-4. The 
document image acquired by the camera system may be combined with one or more other 
documents or multimedia annotations. Robinson stores the image in the registry. 

It would have been obvious to one of ordinary skill in the art at the time the invention 
was made to have combined Robinson into Schena to have created the claimed invention. It 
would have been obvious and desirable to have used the electronic multimedia document 
creation ability of Robinson to have created an electronic multimedia document corresponding to 
the paper multimedia document so that the system could have had an immutable copy of the 
multimedia document as is taught by Robinson in case the paper multimedia document is 
damaged or destroyed. 
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Regarding dependent claim 39, Schena teaches wherein the multimedia annotation 
associated with the paper multimedia document is represented as a first bar code printed on the 
paper multimedia document, wherein the first bar code encodes an audio sound, wherein the first 
bar code printed on the paper multimedia document is capable of being extracted by scanning 
and decoding the first bar code printed on the paper multimedia document, and wherein the 
extracted first bar code can be played via a multimedia player in col. 1 line 61 - col. 2 line 25 
and col. 3 lines 58-63. 

Regarding dependent claim 40, Schena teaches wherein a location indicator of the 
multimedia annotation associated with the paper multimedia annotation is placed on the paper 
multimedia document, wherein the location indicator indicates where the multimedia annotation 
can be retrieved and played in col. 1 line 61 - col. 2 line 25. 

Regarding dependent claim 41, Schena teaches wherein the location indicator 
comprises a Uniform Resource Locator (URL), and a second bar code, wherein the URL is 
indicated in plain text, and wherein the second bar code represents the URL in col. 3 line 58 - 
col. 4 line 6. 

Regarding dependent claims 45 and 46, Schena teaches capturing an audio sound of the 
multimedia annotation from a user using a microphone of the input device to annotate the paper 
document to create a multimedia paper document in col. 1 line 61 - col. 2 line 25 and col. 3 line 
64 - col. 4 line 6. Schena does not specifically indicate at what point in time or with what event 
the multimedia annotation is created. It would have been obvious to one of ordinary skill in the 
art at the time the invention was made to have modified Schena to have automatically captured 
an audio sound annotation for the paper document when the paper document was being 
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reproduced so that the multimedia annotation is relevant to both the user and the paper 
document. 

Regarding dependent claims 47 and 48, Schena teaches capturing a video clip of the 
multimedia annotation from a user using a video camera of the input device to annotate the paper 
document to create a multimedia paper document in col. 1 line 61 - col. 2 line 25 and col. 3 line 
64 - col. 4 line 6. Schena does not specifically indicate at what point in time or with what event 
the multimedia annotation is created. It would have been obvious to one of ordinary skill in the 
art at the time the invention was made to have modified Schena to have automatically captured a 
video clip annotation for the paper document when the paper document was being reproduced so 
that the multimedia annotation is relevant to both the user and the paper document. 

Regarding dependent claim 49, Schena does not teach in response to a request to 
retrieve a second multimedia document, performing a content-based search for the requested 
multimedia document within the storage based on the content of the multimedia annotation 
associated with the requested multimedia document. Robinson does teach parsing and indexing 
the multimedia documents contained in the registry for retrieval in fig. 1 and the "Registry" 
section in pages 2-3. It would have been obvious to one of ordinary skill in the art at the time the 
invention was made to have combined Robinson into Schena to have created the claimed 
invention. It would have been obvious and desirable to have used the registry index of Robinson 
to have searched and located the appropriate multimedia document to retrieve. 

Regarding dependent claim 50-52, Schena does not teach wherein the content-based 
search is performed by OCR, audio speech recognition, or video face recognition techniques on 
the multimedia annotations of the multimedia documents being searched. Robinson does teach 
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parsing and indexing the contents of the multimedia documents contained in the registry using 
OCR, audio speech recognition, or video face recognition techniques in fig. 1 and the "Registry" 
section in pages 2-3. It would have been obvious to one of ordinary skill in the art at the time the 
invention was made to have combined Robinson into Schena to have created the claimed 
invention. It would have been obvious and desirable to have used the registry index of Robinson 
to have searched and located the appropriate multimedia document to retrieve. 

7. Claims 11, 23, 35, and 43 are rejected under 35 U.S.C. 103(a) as being unpatentable 
over Schena et al. (hereinafter "Schena"), US 6,448,979 Bl filed 1/25/1999 in view of 
Robinson et al. (hereinafter "Robinson"), "The Origami Project: Paper Interfaces to the 
World-Wide Web", 

[http://www.cl.cam.acuk/Research/Origami/Origamil997f/index.html], submitted to 
Webnet 97 in November 1997, pages 1-6 as applied to claims 8, 20, 32, and 38 above, and 
further in view of Cunningham, US 6,208,436 Bl filed 2/9/1998. 

Regarding dependent claims 11, 23, and 35, Schena does not teach automatically 
sending a second multimedia document to a recipient by electronic mail as a part of reproducing 
the paper document via the document reproduction system, wherein the recipient is specified by 
a user via an interface of the document reproduction system when the user reproduces the paper 
document using the document reproduction system. Cunningham does teach automatically 
sending a second document to a recipient by electronic mail as a part of reproducing the paper 
document via the document reproduction system, wherein the recipient is specified by a user via 
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an interface of the document reproduction system when the user reproduces the paper document 
using the document reproduction system in fig. 3-4, 6-7, and col. 5 lines 59-65. 

It would have been obvious to one of ordinary skill in the art at the time the invention 
was made to have combined Cunningham into Schena in view of Robinson to have created the 
claimed invention. It would have been obvious and desirable to have used the automatic 
document emailing of Cunningham to have improved Schena in view of Robinson so that the 
multimedia document could have been shared with other users for multi-user document 
collaboration as is taught by Cunningham in col. 4 lines 26-39. 

Regarding dependent claim 43, Schena does not teach automatically sending the 
electronic multimedia document to a recipient as a part of reproducing the paper document via 
the document reproduction system, wherein the recipient receives the electronic multimedia 
document in the form of an attachment to an electronic mail, wherein the recipient is specified by 
a user via an interface of the document reproduction system when the user reproduces the paper 
document using the document reproduction system. Cunningham does teach automatically 
sending the electronic multimedia document to a recipient as a part of reproducing the paper 
document via the document reproduction system, wherein the recipient receives the electronic 
multimedia document in the form of an attachment to an electronic mail, wherein the recipient is 
specified by a user via an interface of the document reproduction system when the user 
reproduces the paper document using the document reproduction system in fig. 3-4, 6-7, and col. 
5 lines 59-65. 

It would have been obvious to one of ordinary skill in the art at the time the invention 
was made to have combined Cunningham into Schena in view of Robinson to have created the 
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claimed invention. It would have been obvious and desirable to have used the automatic 
document emailing of Cunningham to have improved Schena in view of Robinson so that the 
multimedia document could have been shared with other users for multi-user document 
collaboration as is taught by Cunningham in col. 4 lines 26-39. 

8. Claims 12, 24, and 36 are rejected under 35 U.S.C. 103(a) as being unpatentable 
over Schena et al. (hereinafter "Schena"), US 6,448,979 Bl filed 1/25/1999 in view of 
Robinson et al. (hereinafter "Robinson"), "The Origami Project: Paper Interfaces to the 
World-Wide Web", [http://www,cLcam.ac.uk/Research/Origami/Origamil997f/index.html] 
and Cunningham, US 6,208,436 Bl filed 2/9/1998 as applied to claims 11, 23, and 35 above, 
and further in view of Halliday et ah, US 5,880,740 filed 7/12/1996. 

Regarding dependent claims 12, 24, and 36, Schena does not teach wherein the 
recipient receives the image of the paper document and the multimedia annotation in the form of 
Multi-purpose Internet Mail Extension (MIME). Halliday does teach sending an image of a 
document in the form of Multi-purpose Internet Mail Extension (MIME) in col. 8 lines 5-28. It 
would have been obvious to one of ordinary skill in the art at the time the invention was made to 
have combined Halliday into Schena in view of Robinson and Cunningham to have created the 
claimed invention. It would have been obvious and desirable to have used the Multi-purpose 
Internet Mail Extension (MIME) teaching of Halliday to have implemented the automatic 
sending of the electronic multimedia document of Schena in view of Robinson and Cunningham 
so that the recipient would have used any common email client to have received the electronic 
multimedia document sent from the user. 
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Response to Arguments 
9. Applicant's arguments filed 3/10/2005 have been fully considered but they are not 
persuasive. Regarding Applicant's argument in pages 14 and 15 that Schena et al. (hereinafter 
"Schena") does not disclose the limitations of independent claim 1, the Examiner respectfully 
disagrees. Schena discusses in the abstract that the machine readable code on the paper is used 
to communicate corresponding multimedia information when the machine readable code is read 
by a scanner. The URL encoded in the machine readable code points to a multimedia file. Thus, 
the machine readable code on the printed medium annotates the printed medium with the 
referenced multimedia file, thus combining the printed medium with virtual multimedia. In 
combining the multimedia presentation with the printed medium, Schena has created a 
multimedia document of the claimed invention. Schena discusses in col. 2 lines 6-8 that the 
multimedia annotation may contain one or more of textual, audio, or video information. Thus, 
the Examiner maintains that Schena discloses all the limitations of independent claims 1,13, 25, 
and the argued limitations of independent claim 37. 

Regarding Applicant's argument in pages 16 and 17 that Schena and Robinson et al. 
(hereinafter "Robinson") do not teach all the limitations of dependent claims 8, 20, and 32, the 
Examiner respectfully disagrees. Robinson teaches in that the document is unconsciously 
captured and stored in the registry. Robinson notes in page 4, paragraphs 1-6 that the camera 
system captures the image of the document and location of the URL annotations, which can have 
interactors associated with those locations. Schena also contains teachings of multimedia 
annotations on the printed medium. Since Robinson teaches unconsciously generating an image 
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of a paper document including annotations and Schena teaches that the annotations may be 
multimedia annotations, thus forming a multimedia document, the Examiner maintains that 
Schena and Robinson in combination teach all the limitations of dependent claims 8, 20, and 32. 

Regarding Applicant's argument in pages 17 and 18 that Schena and Robinson do not 
teach the limitations of dependent claims 45-52, the Examiner respectfully disagrees. In 
response to applicants argument that the examiner's conclusion of obviousness is based upon 
improper hindsight reasoning, it must be recognized that any judgment on obviousness is in a 
sense necessarily a reconstruction based upon hindsight reasoning. But so long as it takes into 
account only knowledge which was within the level of ordinary skill at the time the claimed 
invention was made, and does not include knowledge gleaned only from the applicant's 
disclosure, such a reconstruction is proper. See In re McLaughlin, 443 F.2d 1392, 170 
USPQ 209 (CCPA 1971). In the case of combining the teachings of Schena and Robinson, 
Robinson teaches parsing and indexing the multimedia documents contained in the registry for 
retrieval in fig. 1 and the "Registry" section in pages 2-3 and the Examiner believes this provides 
a suggestion to one of ordinary skill in the art to have modified Schena to have a performed a 
search up on an index of stored multimedia such as the registry taught by Robinson. 

For these reasons, the Examiner maintains that the invention as claimed is either taught or 
suggested by the cited prior art in the various combinations set forth. 
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Conclusion 

10. THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time 
policy as set forth in 37 CFR 1.136(a). 

A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within TWO 
MONTHS of the mailing date of this final action and the advisory action is not mailed until after 
the end of the THREE-MONTH shortened statutory period, then the shortened statutory period 
will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 
CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, 
however, will the statutory period for reply expire later than SIX MONTHS from the mailing 
date of this final action. 

1 1 . Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Peter J Smith whose telephone number is 571-272-4101. The 
examiner can normally be reached on Mondays-Fridays 7:00am-3 :30pm. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Joseph H Feild can be reached on 571-272-4090. The fax phone number for the 
organization where this application or proceeding is assigned is 703-872-9306. 



Application/Control Number: 09/526,031 



Page 16 



Art Unit: 2176 

Information regarding the status of an application may be obtained from the Patent 
Application Information Retrieval (PAIR) system. Status information for published applications 
may be obtained from either Private PAIR or Public PAIR. Status information for unpublished 
applications is available through Private PAIR only. For more information about the PAIR 
system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR 
system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). 



PJS 

5/10/2005 



SUPERVISORY PATENT EXAMINER 




